Gaze initiated interaction technique

ABSTRACT

A gaze initiated interaction technique for facilitating interaction between wearers of head-mountable devices. Methods are shown which allow users to interact with each other following an event of interlocking gaze. Methods are also shown which allows the operator of a personal point-of-interest beacon to control and limit interaction from the public thus ensuring a greater degree of privacy. Methods are also introduced which allow a mode of interaction to be determined by a remote service after mutual authorization of each interacting user. This disclosure also introduces a system of a head-mountable device which comprises a directional beacon transceiver system using beams in the infrared light spectrum enabling the detection of a gaze interlock event.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of the following U.S. patentapplication, which is incorporated by reference in its entirety:

U.S. Non-Provisional application Ser. No. 17/013,604, filed Sep. 6,2020, which is a continuation of the following U.S. patent application,which is incorporated by reference in its entirety:

U.S. Non-Provisional application Ser. No. 16/406,954, filed May 8, 2019,which is a continuation of the following U.S. patent application, whichis incorporated by reference in its entirety:

U.S. Non-Provisional application Ser. No. 15/017,606, filed Feb. 6,2016.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable to this application.

BACKGROUND Field

Example embodiments in general relate to a gaze initiated interactiontechnique for inviting, initiating and performing interaction betweenwearers of head-mountable devices.

Prior Art

Any discussion of the related art throughout the specification should inno way be considered as an admission that such related art is widelyknown or forms part of common general knowledge in the field.

With the advent of wearable devices the concept of point-of-interestbeacons has received an increasing amount of attention in recent years,with the iBeacon technology becoming more and more prevalent. Currenttechnology is centered around beacons being assigned unique identifiers,such as UUIDs, and configuring these beacons to transmit their staticand permanent identifiers with a low range omnidirectional transmitter,such as Bluetooth LE. Whilst this technology is particularly suited toallow interfacing the point-of-interface beacons of large and inanimateobjects, such as buildings or storefronts, there are a number ofshortcomings which currently stand in the way of using point-of-interestbeacons to facilitate interpersonal communication.

SUMMARY

This disclosure provides, in part, a group of methods.

One method includes detecting the gaze of a person wearing ahead-mounted device and providing means for interacting with thatperson. Several means are known that allow detecting beacons associatedwith points-of-interest in an augmented reality environment. However,when a point-of-interest is associated with a person that person may notdesire being tracked or contacted by an unrestricted group of people.The method also provides a means to authenticate against apoint-of-interest beacon through mediation of a remote network, such asa social network.

Another method includes enabling the user of a head-mountable device toinvite interaction from other members of a shared network. The methodincludes providing a means for the beacon user to restrict visibilityand interaction with regards to other users. Therefore the beacon userremains anonymous to other users of similar head-mountable devices andonly becomes visible to a select group of members of a network. Themethod includes a means to configure the head-mountable device toproject the beam of the point-of-interest beacon in the direction of theuser's gaze.

Another method includes configuring a remote service to provideauthorization for interaction with a point-of-interest. The serverdetermines whether access should be granted based on inclusion orexclusion criteria set in user profiles. If access is granted, thefunction returns an interface to the point-of-interest.

This disclosure also provides, in part, a system of a head mounteddevice (HMD). The system includes a directional beacon transmitting aunique identifier. The unique identifier may be changed in regularintervals such as to avoid tracking and to provide a degree of privacyand anonymity. The system may implement the directionalpoint-of-interest beacon with an infrared light source. The system alsoincludes a means for receiving a directional point-of-interest beacon,which may be implemented by an infrared-spectrum camera pointing in thesame direction as the gaze of the user.

There has thus been outlined, rather broadly, some of the features ofthe gaze initiated interaction technique in order that the detaileddescription thereof may be better understood, and in order that thepresent contribution to the art may be better appreciated. There areadditional features of the gaze initiated interaction technique thatwill be described hereinafter and that will form the subject matter ofthe claims appended hereto. In this respect, before explaining at leastone embodiment of the gaze initiated interaction technique in detail, itis to be understood that the gaze initiated interaction technique is notlimited in its application to the details of construction or to thearrangements of the components set forth in the following description orillustrated in the drawings. The gaze initiated interaction technique iscapable of other embodiments and of being practiced and carried out invarious ways. Also, it is to be understood that the phraseology andterminology employed herein are for the purpose of the description andshould not be regarded as limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments will become more fully understood from the detaileddescription given herein below and the accompanying drawings, whereinlike elements are represented by like reference characters, which aregiven by way of illustration only and thus are not limitative of theexample embodiments herein.

FIG. 1A is a block diagram of an example embodiment.

FIG. 1B is a first perspective view of a gaze initiated interactiontechnique in accordance with an example embodiment.

FIG. 2 is a frontal view illustrating a technique for utilizinginterlocking gaze in respect of wearers of a head-mountable device.

FIG. 3 illustrates an example embodiment relating to establishingcontact with the user of a personal point-of-interest beacon.

FIG. 4 illustrates an example embodiment relating to running an app on ahead-mountable device relating to communications with the user of apersonal point-of-interest beacon.

FIG. 5 is a block diagram depicting examples of the type of interactionwhich can follow acquisition of a personal point-of-interest beacon.

FIG. 6 is a flow diagram relating to an example embodiment depictingcommunications between two personal point-of-interest beacon users.

FIG. 7 is a block diagram depicting the system architecture of anexample embodiment of a system utilizing a gaze initiated interactiontechnique.

DETAILED DESCRIPTION

Among the techniques known in the prior art are those point-of-interestbeacons which transmit static identifiers, commonly UUIDs (universallyunique identifiers). Even though a UUID can be anonymous at the point ofcreation, once it has been associated with a person it can be used totrack that person, which is considered a disadvantage in any scenarioinvolving a requirement of privacy. The present invention, in part,provides a technique which allows the user of head-mountable device topublicly broadcast a constantly changing identifier thus making itdifficult to track or identify the beacon user. At the same time thebroadcast identifier can serve the function to invite interaction from aselect group of member of a shared network, such as a shared socialnetwork. Importantly the novel technique does not necessarily restrictinteraction to other network users, i.e. social network friends,previously associated with the beacon user. The technique also providesmethods which allow beacon users to restrict access by setting profilefiltering criteria, such as by setting which profile parameters anothersocial network user must exhibit in order to be allowed to interfacewith the beacon user. Therefore the novel technique offers, in part, thepossibility that a social network, for example, can mediateauthentication and authorization between beacon users. Other thanauthentication and authorization, the technique can also serve to selectand serve the appropriate interface for interaction. There may be anumber of augmented-reality apps, which can provide such interface. Thedisclosed technique will allow a social network to select the mostappropriate interface based on an algorithm taking into account bothuser profiles and the context of a gaze interlock event. The techniquewhich is subject to this invention also proposes a system of ahead-mountable device comprising a beacon system, whereby both thetransmitter and the receiver beam are envisaged to be directional,whereas prior art embodiments of such beacons tend to beomnidirectional. The reason for seeking directional beams lies insolving the problem of how find a technological equivalent to the basichuman interaction of the gaze of two persons meeting. In a preferredembodiment the directional beacon transmitter is implemented as adirectional beam of light in the infrared spectrum, whereas thedirectional beacon receiver is implemented as an infrared-spectrumcamera facing the same direction as the gaze of the user of ahead-mountable device.

The block diagram of FIG. 1A illustrates the process flow of a beaconinteraction from the perspective of the user of a head-mounted devicelooking to interact with other users. The process starts 1000 with apersonal point-of-interest beacon being acquired 1001. At this point theuser has only an anonymous identifier to go by and requiresauthorization 1002 before the anonymous identifier can be used for anykind of purpose. The authorization is requested from a remote serverwhich can look up the anonymous identifier and match it with a profileon a network, such as a social network. As long as the authorizationprocess is successful, the remote server returns a handle to aninterface 1003. Once an interface to the point of interest is availablethe user is given the option to commence interaction with thepoint-of-interest 1004. If the user accepts that invitation forinteraction 1005, then an application is started which is capable ofusing the interface returned in step 1003. The application is executeduntil a disengagement event 1007 is received, which ends 1008 theinteraction.

The schematic diagram in FIG. 1B depicts the hardware required in thescenario of a preferred embodiment of the present invention. Bothpersons taking part in the interaction are wearing a head-mountabledevice 140. The head-mountable device comprises both a beacontransmitter 110 and a beacon receiver 120. In a preferred embodimentboth the beacon transmitter and the beacon receiver are configured tooperate with light of the infrared spectrum. Both the transmitting andreceiving beams are configured to coincide with the gaze of the user130. Through the medium of the beacon beams two functions are beingimplemented. One such function is the exchange of the describedanonymous unique identifiers 180. The other function is to track thepoint-of-interest beacon 175 in order to be able to project thecorresponding beacon user 170 as an augmented reality target. In thepreferred embodiment both interacting users 170 175 use a system whichcombines a head-mountable device with a smartphone-type handset 150. Itis envisaged that all distributed functions with the exception of saidbeacon tracking and UUID exchange are processed by making use of cloudservices 160 which are in turn accessed by functionality provided by therespective smartphone-type handsets 150.

FIG. 2 illustrates the gaze initiated interaction technique in thecontext of an augmented-reality setting whereby the figure depicts thefirst step of the technique comprising scanning the environment forother personal point-of-interest beacons in the field of view. The user250 of a head-mountable device gazes in the direction of a crowd ofpeople 235 whereby in the crowd there is a person 230 who operates asimilar head-mountable device comprising a point-of-interest beacon 220.The beacon 220 disseminates a beam 210 corresponding to the gaze of theuser 230. The first user 250 operates a similar beam transceivercorresponding to her gaze 240. Once an intersection of gaze-relatedbeams 260 takes place, unique identifiers can be exchanged. In thedepicted embodiment, however, it is sufficient if one of the users picksup the unique identifier of the beacon of the other user.

FIG. 3 illustrates the gaze initiated interaction technique from thepoint of view of a user 250 at a point in time when authorization hasbeen successfully performed on a plurality of social networks. The frame305 depicts the boundary of the augmented-reality scene depicted. Theframe contents simulate what the user 250 may see. Frame 350 denotes amenu panel superimposed on the augmented reality scenery. Frame 340denotes an individual selectable menu item. The scenery comprises acrowd of people which are not enhanced by augmented reality. There isone individual 310 in the crowd who is enhanced in the augmented realityenvironment. The enhancement consists of providing a means of tracking320 the individual in the field of view. An icon 330 hovers above thetracked individual 310. The said icon denotes that the trackedindividual 310 offers an invitation for interaction to the other user250. At this point of the interaction process no interface forinteraction is available yet. In the example the user 250 is notifiedthat authorization has been granted for interaction with thepoint-of-interest by altogether three social networks 361 362 363. Thesaid user is prompted to select over which one of the social networksoffered as selection the proposed interaction is to take place. Theselection of network is important for the reason that the interfaceoffered for interaction will normally be different for each network. Iffor example, the point-of-interest grants authorization both forinteraction over a social network designed for dating and a socialnetwork designed for professional networking, then it is to be expectedthat the corresponding interfaces are substantially different.

FIG. 4 depicts the use case started in FIG. 3 at the next phase ofinteraction. At this stage a preferred social network has been selected,the corresponding interface has been retrieved and an application hasbeen launched which makes use of the said interface. Whilst theapplication is active a suitable icon 400 is projected above the trackeduser 310 denoting that the currently active application interacts withsaid tracked user. Menu card 410 denotes an arbitrary interactionoption, in this particular example the option to send a text message tothe tracked user 310.

FIG. 5 is a block diagram providing more detail on preferred embodimentsof the type of interaction between users of the gaze initiatedinteraction technique. Once an interface is obtained enabling contact toa personal point-of-interest beacon an application 500 is launched. Themost basic application of that type would be one in which the seconduser is added as a friend 501 on the social network which has mediatedthe described authorization. Another possible application is theexchange of business cards, such as VCARDs 502. Sending a short textmessage 503 to the second user is another possible example application.In a similar way two way voice communications 504 may be initiated. Theapplication may also restrict interaction simply to tracking the seconduser as an augmented-reality target 505. This use case would beparticularly interesting in keeping track for example of a tour guide ina very crowded place. Another application would be to simply retrieveand browse profile attributes of the second user 506. In a gamingscenario it would be of particular interest to utilize the describedgaze initiated interaction technique to initiate an application whichallows to introduce a previously not acquainted member of the publicinto a running instance of an augmented reality game.

FIG. 6 describes an end-to-end use case of a gaze-initiated interactionbetween a first user 250 and a second user 310 in an augmented-realityenvironment. The second user 310 is operating a personalpoint-of-interest beacon transmitting an anonymous temporary uniqueidentifier which is only known to one social network. Both the first andthe second user use head-mountable devices 695 in combination withsmartphone-type handheld mobile devices 150. The interaction starts withthe first user 250 detecting the point-of-interest beacon 601 of thesecond user 310. Upon detection of the point-of-interest beacon theunique identifier of the beacon is retrieved. It has to be noted that apoint-of-interest beacon may transmit more than one identifier. If thatis the case, the following lookup process is repeated for each of theidentifiers. In the described use case, however, it is assumed that onlyone unique identifier is being transmitted. For the next step thehandset 150 of the first user 250 establishes a connection with theInternet in order to access a plurality of cloud services. From thepoint of view of the first user the input 620 for the lookup process isthe unique ID 698 of the beacon relating to the second user, whereas theoutput 625 is a service interface to the second user. The steps requiredto retrieve the service interface based on the lookup of the uniqueidentifier are as follows: at first the identifier is submitted 630 to alookup service of a first social network 670, which is a social networkon which the first user 250 has an account and a profile; if the uniqueidentifier 698 is registered on that social network an algorithm 685comprising a matching logic is executed, in such a way that thealgorithm takes into account inclusion and exclusion criteria associatedwith both the profile of the first user 250 and the second user 310; theoutcome of the matching process determines whether authorization isgranted to the interaction request; in the present use case lookupprocess 630 returns a negative authorization result 635; the next stepscomprise iterating through a plurality of social networks to which thefirst user 250 has subscribed; for each social network the lookupprocess is repeated; in the specific use case lookup 640 is successfulsince the matching algorithm 690 yields positive authorization and as aresult the social network 675 grants authorization for interaction 645;once authorization has been obtained it is necessary to obtain anapplication for interaction, for that purpose the social network 675 isqueried for the identifier of a suitable application; subsequently anidentifier for the respective application is obtained 655; in order tobe able to run the application instance data has to be obtained 657,also from social network 675; before the application can be launched itis necessary to ensure that the application itself is available on themobile devices of the first user 250; in the present use case theapplication has not yet been installed and has to be requested 660 anddownloaded from an online app repository 680. Once authorization hasbeen granted and the application is running and has been provided withinstance data, then the user 250 is notified that augmented-realityinteraction with user 310 is possible through application instance 606.Both users 250 310 are queried to agree to a virtual handshake whichinitiates interaction through the app 606, if affirmed. For as long asthe application 606 is active both users track each other in theaugmented-reality environment 615.

FIG. 7 depicts the architecture of an example embodiment of a systemimplementing aspects of the novel gaze initiated interaction technique.The system comprises two mobile devices, namely a head-mountable device740 and a smartphone-type mobile device 745. Both devices 740 745 areused in conjunction with each other and are thus treated as a singleunit for the purposes of the system diagram. The system comprises one ormore processing units 705, a transceiver 710 for mobile communicationsvia the smartphone-type device 745 and a plurality of communicationinterfaces 715. The device comprises a data storage component 739, whichin turn comprises stored program logic 730 and stored program data 735.The system also comprises a plurality of user input and output devices270. The plurality of input and output devices comprises a beaconprocessor 759. The beacon processor comprises a directional beacontransmitter 751, a directional beacon receiver 753 and a beacon tracker753. Other user input and output devices in the preferred embodiment area front-facing camera 761, a display 762, a speaker 763, a microphone764 and a touchpad 765.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Although methods and materialssimilar to or equivalent to those described herein can be used in thepractice or testing of the wink gesture based control system, suitablemethods and materials are described above. All publications, patentapplications, patents, and other references mentioned herein areincorporated by reference in their entirety to the extent allowed byapplicable law and regulations. The gaze initiated interaction techniquemay be embodied in other specific forms without departing from thespirit or essential attributes thereof, and it is therefore desired thatthe present embodiment be considered in all respects as illustrative andnot restrictive. Any headings utilized within the description are forconvenience only and have no legal or limiting effect.

What is claimed is:
 1. A method for processing operations relating togaze-interlock events on a social network server, the method comprising:registering a first unique identifier as an attribute against a firstuser profile on a social network, wherein said first user profilebelongs to a person; and receiving a second unique identifier from amobile device configured for detecting gaze-interlock events, wherein agaze-interlock event is detected when a field of view of said person anda field of view of a human operator of said device are overlapping, andwherein said second unique identifier is determined as part of thedetection of said event; and receiving a user identifier from saidmobile device, wherein said user identifier is linked to said humanoperator and wherein said user identifier is linked to a second socialnetwork profile; and sending instructions for tagging said person inaugmented reality to said mobile device in response to receiving saidsecond unique identifier, wherein said response is conditional on:finding the first social network profile through a lookup procedure,wherein said lookup procedure comprises using said second uniqueidentifier as a key and wherein the first social network profile isfound when said second unique identifier matches said attribute.
 2. Themethod of claim 1 further comprising: executing a first matching logic,wherein said first matching logic is configured to compare one or moreexclusion or inclusion criteria with said second profile, wherein saidcriteria are associated with said first profile.
 3. The method of claim2 further comprising: executing a second matching logic, wherein saidsecond matching logic is configured to compare one or more exclusion orinclusion criteria with said first profile, wherein said criteria areassociated with said second profile.
 4. The method of claim 3 furthercomprising: evaluating the result of one or more matching logics andauthorizing the tagging of said person in the augmented reality of saidhuman operator based on said result.
 5. The method of claim 4 furthercomprising: sending instance data to said mobile device, wherein saidsending of instance data is conditional on said result and wherein saidinstance data comprises data which is required in order for said personand said human operator to be able to interact through an applicationfor social networking.
 6. The method of claim 5 wherein said instancedata enables said application to offer adding said person as a socialnetworking friend.
 7. The method of claim 5 wherein said instance dataenables said application to offer messaging said person.
 8. The methodof claim 5 wherein said instance data enables said application to offerincluding said person in an augmented reality game.
 9. The method ofclaim 5 wherein said instance data enables said application to offerbrowsing information relating to said person.
 10. The method of claim 5wherein said instance data enables said application to offer opening atwo-way live communication channel with said person.
 11. The method ofclaim 1 further comprising: sending instructions for populating aplurality of menu cards to said mobile device, wherein said menu cardsoffer a choice of options for interacting through social networks.
 12. Adevice for social networking in augmented reality, the devicecomprising: one or more processing units; and a camera for capturingoptical data, wherein said camera is configured to be pointing in adirection which is comprised in a field of view of a human operator ofsaid device and wherein said optical data, when processed by the one ormore processing units, is capable of yielding an identifier as a resultof said human operator being in the field of view of a person; and adisplay, wherein said display is capable of superimposing augmentedreality content over a real-word scene; and a transceiver for mobilecommunications; and wherein said one or more processing units areconfigured in such a way that, contingent on the detection of saididentifier: said processing units cause control signals to be relayed tosaid transceiver for querying one or more social networks with saididentifier and wherein said device causes said person to be tagged inaugmented reality and wherein said tagging is conditional on a responseto said querying and wherein said tagging is executed by said processingunits relaying control signals to said display.
 13. The device of claim12 wherein said camera is an infrared-spectrum camera.
 14. The device ofclaim 12 wherein said tagging comprises a menu panel which issuperimposed on the augmented reality scenery.
 15. The device of claim14 wherein said menu panel displays menu cards and wherein said menucards offer modes for socially networking with said person.
 16. Thedevice of claim 14 wherein said menu panel comprises a plurality of menucards, wherein each menu card is associated with a social network andwherein the plurality of menu cards comprises associations with aplurality of social networks.
 17. The device of claim 12, wherein saididentifier is anonymous in such a way that, when viewed in isolation orwhen used as a key for querying a public data store, said identifier isunsuitable for revealing the identity of said person.
 18. The device ofclaim 12, wherein said identifier is person-specific in such a way thatthe identifier is a unique identifier which has been registered againsta social network profile of said person.
 19. The device of claim 12,wherein said camera is forward-facing.
 20. The device of claim 19,wherein said camera is affixed to a head-mountable component of saiddevice.
 21. The device of claim 12 further comprising a directionalbeacon transmitter.
 22. The device of claim 21 wherein said directionalbeacon transmitter is configured to emit a beam of light, wherein awavelength of said light is comprised in the infra-red spectrum.
 23. Thedevice of claim 22 wherein said directional beacon transmitter isconfigured to direct said beam of light in a direction which iscomprised in said field of view of said human operator and wherein awavelength of said light is comprised in the infra-red spectrum andwherein said beam is modulated in such a way that it carries an outboundidentifier.
 24. The device of claim 12 wherein said camera is configuredto image light, wherein said light is of a type which is emitted by adirectional beacon transmitter, wherein said transmitter emits one ormore beams of infra-red light.
 25. A non-transitory computer readablestorage medium having data stored therein representing softwareexecutable by a computer, the software including instructions to processgaze interlock events, the storage medium comprising: instructions forreceiving event notifications from a detector for gaze-interlock eventswherein said detector comprises a camera and wherein said events aretriggered when the gaze of a person is being directed towards a humanoperator of said camera and when said camera is pointing in a direction,wherein said direction is comprised in the field of view of saidoperator; and instructions for tagging said person in the augmentedreality of said operator, contingent on the detection a gaze-interlockevent, wherein said tagging is conditional on the responses receivedfrom one or more queries to one or more social networks, wherein saidqueries are parametrized with an identifier and wherein said identifieris obtained when detecting the gaze-interlock event.
 26. Thenon-transitory computer readable storage medium of claim 25 wherein saidtagging comprises a menu panel which is superimposed on the augmentedreality scenery.
 27. The non-transitory computer readable storage mediumof claim 26 wherein said menu panel displays menu cards and wherein saidmenu cards offer modes for socially networking with said person.
 28. Thenon-transitory computer readable storage medium of claim 26 wherein saidmenu panel comprises a plurality of menu cards, wherein each menu cardis associated with a social network and wherein the plurality of menucards comprises associations with a plurality of social networks. 29.The non-transitory computer readable storage medium of claim 25, whereinsaid identifier is anonymous in such a way that, when viewed inisolation or when used as a key for querying a public data store, saididentifier is unsuitable for revealing the identity of said person. 30.The non-transitory computer readable storage medium of claim 25, whereinsaid identifier is person-specific in such a way that the identifier isa unique identifier which has been registered against a social networkprofile of said person.